Semi-Supervised Web Wrapper Repair via Recursive Tree Matching

Continuous data extraction pipelines using wrappers have become common and integral parts of businesses dealing with stock, flight, or product information. Extracting data from websites that use HTML templates is difficult because available wrapper methods are not designed to deal with websites that change over time (the inclusion or removal of HTML elements). We are the first to perform large scale empirical analyses of the causes of shift and propose the concept of domain entropy to quantify it. We draw from this analysis to propose a new semi-supervised search approach called XTPath. XTPath combines the existing XPath with carefully designed annotation extraction and informed search strategies. XTPath is the first method to store contextual node information from the training DOM and utilize it in a supervised manner. We utilize this data with our proposed recursive tree matching method which locates nodes most similar in context. The search is based on a heuristic function that takes into account the similarity of a tree compared to the structure that was present in the training data. We systematically evaluate XTPath using 117,422 pages from 75 diverse websites in 8 vertical markets that covers vastly different topics. Our XTPath method consistently outperforms XPath and a current commercial system in terms of successful extractions in a blackbox test. We are the first supervised wrapper extraction method to make our code and datasets available (online here: http://kdl.cs.umb.edu/w/datasets/).